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Abstract Body 


Background / Context: 

Can successful schools replicate? In a climate of school turnarounds, charter conversions, and 
new school openings, an important question is whether schools that have demonstrated success 
with in one or several schools can replicate their success with additional schools. The federal 
government’s Investing in Innovation (i3) grant program has been provided over $650 million to 
scale up successful programs, with two of the largest grants going to school creation and 
replication through the KIPP network of charters schools and school turnaround through the 
whole school refonn efforts Success for All. Of course, the idea of replicating successful schools 
is not a new one, with major (though largely unsuccessful) efforts in the 1990’s to scale up whole 
school reform models to large numbers of schools (Berends, et al. 2002). 

In this paper, we use a change in the Massachusetts law that authorizes charter schools to 
examine whether charters deemed by the state as “proven providers” do indeed have the largest 
impacts on student outcomes and whether their replication campuses are similarly successful. 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this paper is to answer three research questions. First, do the charter schools that 
are allowed to replicate under Massachusetts law have larger impacts than other charter schools? 
In other words, are “proven providers” indeed proven? Second, do the new replication campuses 
produce similar test score gains to their parent campuses? And finally, how has the demand for 
charter schools changed over the time period when these new schools were introduced? 

Setting: 

In January 2010, Massachusetts’ Governor Patrick signed An Act Relative to the Achievement 
Gap into law in Massachusetts. An ambitious piece of education legislation, several of its 
provisions focused on charter schools. Specifically, the Act increased the cap for charter schools 
in the 10 percent of lowest performing districts in the state from 9 percent to 18 percent of a 
district’s annual budget, by allowing “proven providers” to start new schools or expand 
enrollment. This change in law came at a time when Boston area charters had demonstrated 
impressive score gains. Lottery-based estimates show that each year spent at a charter middle 
school boosts scores by about a fifth of a standard deviation in English Language Arts (ELA) and 
over a third of a standard deviation in math (Abdulkadiroglu et al., 2011). Since that change in 
the law, 13 new charters have opened in Boston, the majority of them replication campuses of 
successful Massachusetts schools that were deemed “proven providers” by the state. 

Population / Participants / Subjects: 

Schools: We study 16 charter middle schools in Boston. This includes four schools designated as 
“proven providers” by the state, seven replicate campuses, and three schools that did not seek to 
replicate, which we call “other charters.” Excluded from the study sample are one middle school 
that closed, one middle school that has not supplied lottery records, and several schools that 
serve middle school grades but do not have an entry lottery in 5 th or 6 th grade. 

Students: There are 7,851 charter middle school lottery applicants in our sample. In the pre- 
expansion years, applicants tend to have higher baseline test scores than the traditional BPS 
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population, and are more likely to be African-American and less likely to be Latino/a. Applicants 
in this period are also less likely to be English Language Learners or qualify for free or reduced 
price lunch. In the post-expansion period, charter applicants tend to be more similar to traditional 
BPS students, with only very small differences in prior test scores, ELL, special education, and 
subsidized lunch status. 

Intervention / Program / Practice: 

A defining feature of Massachusetts' successful urban charter schools appears to be adherence to 
No Excuses pedagogy, an approach to urban education described in a book of the same name 
(Thernstrom and Thernstrom, 2003). No Excuses schools emphasize discipline and 
comportment, traditional reading and math skills, extended instruction time, and selective teacher 
hiring. Massachusetts' No Excuses charters also make heavy use of Teach for America (TFA) 
participants and alumni and provide extensive and ongoing feedback to teachers. Like most 
Boston charter schools, the “proven provider” and replication campuses studied here largely 
identify with the No Excuses approach. Another feature of the Massachusetts charter landscape 
is the state's rigorous charter authorization and monitoring process. To date the state has closed 
12 charter schools after they began operations and an additional 4 schools before they opened 
(Massachusetts Department of Elementary and Secondary Education, 2013). Finally, to our 
knowledge, Massachusetts is unique among states in having a “proven provider” provision in 
their charter law that limits new campuses to those whose operators have demonstrated success 
with past schools. 

Research Design: 

We use lotteries to estimate the effects of charter school attendance on MCAS scores and other 
outcomes. This empirical strategy is motivated by the fact that attending a charter school is a 
choice: the decision to apply to a charter may be correlated with family background, ability, or 
motivation. Comparisons between charter and non-charter students may be biased due to these 
differences. Our lottery-based strategy eliminates selection bias by comparing applicants who are 
offered admission in charter lotteries to applicants not offered admission. Since charter lotteries 
are random, offered and not-offered students are similar with respect to background 
characteristics, including unobserved characteristics, and differences in their subsequent 
outcomes reflect the causal effect of charter admission. The second-stage equation for our 
lottery-based two-stage least squares (2SLS) analysis links charter school attendance with 
outcomes as follows: 

5 

Ji = ^ Sjdij + y'Xi + ^ p k c ik + £ t 

j k= 1 

where y„ is the outcome of interest for student i, X, is a vector including grade-year dummies and 
a set of pre-lottery demographic characteristics (race, special education, limited English 
proficiency, subsidized lunch status, and a female -minority interaction) and c, is an error term. 
The dij are dummy variables for all combinations of charter school lotteries (indexed by j) seen in 
the lottery sample. In what follows, we refer to these combinations as “risk sets.” These are 
included because the application mix detennines the probability of receiving an offer, even when 
offers at each school are randomly assigned. The variables of interest are indicators for 
attendance at each of 5 types of charter schools, indexed by k, each with their own parameter, p, 
which captures the causal effect of charter school attendance at that charter school type. The 5 
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charter types are defined as follows: 

• Proven Provider Pre-201 1 : charter schools later designated by the state as “proven providers” 
after the change in the law, in years 2010 and prior 

• Other Charter Pre-201 1 : charter schools never designated proven providers, in 2010 and prior 

• Proven Provider Post-20 1 1 : charter schools designated by the state as proven providers after 
the change in the law, in years 2011 and following 

• Replicate Post-201 1 : new charter schools opened under the proven provider provision 

• Other Charter Post-201 1: charter schools never designated proven providers that are also not 
replication campuses, in 201 1 and following 

We estimate these effects jointly, instead of in separate samples, as students can apply to 
multiple charter types in a given year. This setup ensures that the counterfactual group is made 
up of non-charter students. 

We use charter offer variables as instruments. The initial offer instrument, Z^i, is a dummy 
variable indicating offers made on the day of the charter school lottery to a particular type of 
charter school, indexed by k. Because some applicants who don't receive offers on lottery day do 
so at a later date when their names are reached on a randomly ordered waitlist, we also code a 
second instrument, denoted ever offer, or Z^. The ever offer instrument indicates applicants who 
receive an offer at any time, whether on lottery day or later, at that type of charter school, again 
indexed by k. Applicants who receive an initial offer thus have both instruments for that type 
switched on, while those who receive later offers without an initial offer have only the ever offer 
instrument switched on for that type. Missing values for any instrument are coded as no offer. 

We have five first stage equations for our 2SLS procedure, as each charter school type k has its 
own first stage as follows: 

Cik — ^ ' Hjdij T P Xi T TCkl^ikl "h ^k2^ik2 T 1]i 
j 

where two separate parameters, 7r fel and n k2 , capture the effects of initial and eventual offers on 
charter attendance at a each charter type k. As in the second stage equation, the first stages 
include risk set controls, grade-year dummies, and baseline demographic characteristics. 

Standard errors are clustered at the grade-school-by-year level. 

Our lottery-based empirical strategy depends critically on the assumption that charter lottery 
offers are randomly assigned. This random assignment balances both observed and unobserved 
characteristics between offered and not-offered students. While we cannot check balance for 
unobserved characteristics, we confirm that lottery winners and losers are similar on observed 
dimensions like race, special education status, and baseline (pre-application) test scores. 
Similarly, there are similar rates of follow-up data for lottery winners and losers. 

Data Collection and Analysis: 

Massachusetts charter schools admit students by lottery when they have more applicants than 
seats. We collected lists of charter school applicants and infonnation on the results of admissions 
lotteries from individual charter schools. These lists were then matched to administrative records 
covering all Massachusetts public school students. Our analysis sample is limited to charter 
applicants who applied for a charter middle school seat from Fall 2002 through Fall 2014. We 
limit our analysis to middle schools, as this is where the majority of school replication occurs. 
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We matched applicant records to administrative data using applicants' names, cohorts, and 
grades of application. Where available, infonnation on date of birth, town of residence, race or 
ethnicity, and gender was used to break ties. Among applicants eligible for our study, 94 percent 
were matched to state data. Applicants were excluded from the analysis if they were disqualified 
from the lottery (these are mostly applicants to the wrong grade). We also omit siblings of 
current charter students, late applicants, and some out-of-area applicants. Students submitting 
charter applications in multiple years appear only once in the sample, with data recorded for the 
first application only. Information on baseline demographics and test scores comes from the most 
recent pre-lottery data available in the state database. 

Findings / Results: 

We find that “proven providers” are indeed proven. Before expansion, as seen in Table 1, proven 
provider schools had impacts of about a third of a standard deviation per year of charter school 
attendance in math, which we find to be significantly larger than the 0. 18 standard deviation (a) 
gains per year in “other charters” before expansion (other charters are charter schools with 
lottery information that did not seek to expand under the new law). Proven providers also had 
larger gains in ELA before charter expansion. After the expansion, proven providers and their 
replication campuses have very similar impacts on student test scores: 0.36a for parent campuses 
and 0.33a for replicate campuses in math; 0. 18a for parent campuses and 0.23a for replicate 
campuses in ELA. There are no significant differences between parent and replicate test score 
gains. We conclude that the proven provider schools were indeed able to replicate successfully. 
Both parent and replicate campuses had larger test score impacts than other charters, where the 
other charter effect was 0.2 la in math and 0.13a in ELA. The differences are not statistically 
significant, though p-values in math come close to conventional significance levels, and with 
fewer years contributing to the analysis post-expansion, we have less statistical power to detect 
differences here. 

The impacts on test scores come during a large increase in charter application (Table 2). Before 
expansion, about 16% of middle school students in Boston applied to charter schools. After the 
expansion, about 40% of middle school students applied. More Boston students enroll in charters 
(due to the expansion in number of seats), however the expansion in seats is not enough to meet 
the expansion in demand. Interestingly, post-expansion, there is a negative relationship between 
demand and test score impacts, likely due to the long waitlists at “other charters” and the 
existence of replication campuses absorbing some of the excess demand for the parent campuses. 

Conclusions: 

Our findings suggest that successful school replication is possible, and that Massachusetts’ 
“smart-cap” charter law is one way to regulate charters to promote schools with positive 
academic impacts for students. Additionally, as charter schools have expanded in Boston, so too 
has demand for those schools. At one time, charter school applicants tended to be positively 
selected when compared to traditional BPS students. Post-expansion, there are few differences 
between the groups and little scope for the argument that charters are “cream-skimming.” As we 
continue this project, we will explore the mechanisms that underlie successful replication, 
including tracking how human capital is used in expansion campuses. 
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Appendix B. Tables and Figures 


Table 1: Charter Effects on Test Scores Before and after Charter Expansions 


Pre-Expansion 



Post-Expansion 


Non-Charter 

2SLS 


Non-Charter 


2SLS 


Mean 

Proven 

Other 

Mean 

Proven 

Replicate 

Other 


Providers 

Charters 


Providers 

Campuses 

Charters 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

Math 0.226 

0.334*** 

0.180*** 

0.328 

0.362*** 

0.325*** 

0.211*** 


(0.041) 

(0.029) 


(0.083) 

(0.082) 

(0.062) 

P-value: Equals proven provider 


0.000 



0.715 

0.116 

P-value: Replicate equals other 





0.170 


N 



19245 




ELA 0.233 

0.140*** 

0.089*** 

0.305 

0.183** 

0 229*** 

0.134** 


(0.038) 

(0.026) 


(0.079) 

(0.084) 

(0.056) 

P-value: Equals proven provider 


0.192 



0.646 

0.572 

P-value: Replicate equals other 





0.233 


N 



19168 





Notes: This table reports 2SLS estimates of the effects of time spent in charter schools on test scores. The endogenous 
variables are years spent at each type of charter schools, and the instruments is a a lottery offer and a waitlist offer dummy 
for each school type. Means are for non-charter students who applied to the lotteries. MCAS scores normalized to Boston 
by grade and year. Sample restricted to students with baseline demographic controls who attended BPS or a Boston charter 
school at baseline. All models control for race, sex, special education, limited English proficiency, and subsidized lunch 
status. Year of birth, year of test, and risk set dummies are also included. Middle school regressions pool post-lottery 
outcomes from 5th through 8th grade and cluster by student identifier as well as school-grade-year. 

*significant at 10%; ^significant at 5%; ***significant at 1% 
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Table 2: Application, Offer, and Take-up Rates 



% of Boston 

Students 
Applying to 
Charters 
(1) 

% of Boston 

Students 
with Lottery 
Offers 
(2) 

% of Boston 

Students 
with Lottery 
or Waitlist 

Offers 

(3) 

% of Boston 

Students 
Enrolling in 
Charters 
(4) 

Applicants 
Per Seat 
(5) 



Panel A: Pre-Expansion 



Any Charter 

0.164 

0.054 

0.105 

0.141 

3.024 

Parent Campus 

0.098 

0.026 

0.065 

0.049 

3.821 

Other Charter 

0.092 

0.030 

0.051 

0.046 

3.027 



Panel B: Post-Expansion 



Any Charter 

0.403 

0.098 

0.168 

0.214 

4.110 

Parent Campus 

0.177 

0.039 

0.075 

0.067 

4.530 

Replicate Campus 

0.183 

0.059 

0.114 

0.048 

3.095 

Other Charter 

0.174 

0.026 

0.034 

0.044 

6.586 


Notes: This table shows charter school application, offer, and attendance rates by cohort. 
Cohorts are defined using as students who are present in BPS or Boston charter schools in 
both 4th and 6th grades. In Panel A the sample is charter applicants from 2008 and 2009, in 
panel B the sample is applicants from 2011-2013. 
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